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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )S Responsive to communication(s) filed on 23 September 2003 . 
2a)[3 This action is FINAL. 2b)Q This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 1 1 , 453 O.G. 213. 
Disposition of Claims 

4) ^ Claim(s) 1-17 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) Q Claim(s) is/are allowed. 

6) ^ Claim(s) 1-17 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)S The drawing(s) filed on 01 February 2000 is/are: a)S accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
11 )□ The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a)n All b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 

1 ) □ Notice of References Cited (PTO-892) 4) D Interview Summary (PTO-41 3) Paper No(s). . 

2) d Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) C] Notice of Informal Patent Application (PTO-152) 

3) O Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) d Other: 
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DETAILED ACTION 



1. 



This action is responsive to communications: Request for reconsideration of rejections 



filed on 09/23/2003, to the original application filed on 02/01/2000. 



2. 



Claims 1-17 are pending in the case. Claims 1, 8, and 10 are independent claims. 



Claim Rejections - 35 USC § 103 



3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 



4. Claims 1-17 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Srivastava et al., US 6,549,922 Bl filed 10/01/1999 in view of Sidana, US 6,571,295 Bl 
priority filed 01/31/1996. 

Regarding independent claim 1, Srivastava teaches a means for analyzing a web page 
to identify at least one markup tag to extract metadata from the multimedia file in Fig. 1. 
Srivastava also teaches a means for selecting and executing a media processing program for 
analyzing the content of the multimedia data supplied by a resource to generate metadata 
describing its content in Fig. 1 and col. 2 lines 45-50. Srivastava teaches a means for formatting 
the metadata into a character-based text annotation in Fig. 2 col. 4 lines 8-13. Srivastava also 
teaches indexing which can be used for searching in col. 8 lines 48-52. What Srivastava does not 
teach is a means for combining the web page with the annotation to form an enhanced web page. 
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Sidana does teach a means for combining the web page with the annotation to form an 
augmented (same thing as enhanced) web page in Fig. 5 and col. 2 lines 13-30. It would have 
been obvious to one of ordinary skill in the art at the time the invention was made to combine 
Sidana into Srivastava to create the claimed invention. It would have been desirable to combine 
both the web page and the annotations into one file which can then be conveniently viewed by 
the user. This would facilitate the user's increased understanding of the content contained in the 
web page. 

Regarding dependent claim 2, which is dependent on claim 1, Srivastava and Sidana 
teach the limitations of claim 1 as explained above. Srivastava teaches in col. 4 lines 32-48 a 
means for determining the particular data type of the multimedia supplied by the given resource 
and also a means for selecting a processing program for analyzing the multimedia data formatted 
in accordance with said particular data type. 

Regarding dependent claim 3, which is dependent on claim 1, Srivastava and Sidana 
teach the limitations of claim 1 as explained above. Srivastava teaches in the abstract a means 
for generating text data annotation expressed in accordance with the Extensible Markup 
Language. 

Regarding dependent claim 4, which is dependent on claim 1, Srivastava and Sidana 
teach the limitations of claim 1 as explained above. Srivastava teaches in Fig. 1 and col. 2 lines 
50-56 a means for acquiring metadata which describes the multimedia data supplied by a given 
resource. In Fig. 2 and col. 4 lines 8-13 Srivastava also teaches a means for including the 
additional metadata in a character-based text annotation. 
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Regarding dependent claim 5, which is dependent on claim 4, Srivastava and Sidana 
teach the limitations of claim 4 as explained above. Srivastava teaches that the multimedia file 
displayed in Fig. 1 which will obviously be contained within a markup tag on an internet web 
page and further in col. 2 lines 48-50 that some of the additional data includes information 
obtained from the markup tag. 

Regarding dependent claim 6, which is dependent on claim 4, Srivastava and Sidana 
teach the limitations of claim 4 as explained above. Srivastava teaches in Fig. 1 that a given 
resource may be accessed through the operating system of a computer and that some of the 
additional data may include information obtained from the operating system. 

Regarding dependent claim 7, which is dependent on claim 4, Srivastava and Sidana 
teach the limitations of claim 4 as explained above. Srivastava teaches in Fig. 1 that some of the 
additional information may be obtained via the internet. 

Regarding independent claim 8, Srivastava teaches a parser for identifying markup tags 
in web pages in col. 2 lines 46-48. Srivastava also teaches a processing means for analyzing the 
content of multimedia data identified by the mark up tags to generate metadata describing the 
multimedia data in Fig. 1 and col. 2 lines 46-50. Srivastava teaches a means for translating the 
metadata into a character-based text annotation describing the multimedia data in Fig. 2 and col. 
4 lines 8-13. What Srivastava does not teach is a means for storing the combination of the web 
page and annotation to form an enhanced web page suitable for processing by text-based 
indexing and searching. Sidana does teach a means for storing the combination of the web page 
and annotation to form an enhanced web page suitable for processing by text-based indexing and 
searching in Fig. 5 and col. 2 lines 13-30. 
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It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sidana into Srivastava to create the claimed invention. It would have been 
desirable to combine the annotation with the web page from which the multimedia file came to 
create an augmented, or enhanced, web page which could be more easily understood by the user. 
This enhanced web page would decrease the amount of time required for the user to get the 
information from the web page which they need. 

Regarding dependent claim 9, which is dependent on claim 8, Srivastava and Sidana 
teach the limitations of claim 8 as explained above. Srivastava teaches in the abstract that the 
text annotation is expressed in the Extensible Markup Language. 

Regarding independent claim 10, Srivastava teaches in Fig. 1 and col. 2 lines 46-48 
identifying one or more mark up tags in a web page which respectively identify one or more 
external resources which provide multimedia data. Srivastava also teaches in Fig. 1 and col. 2 
lines 46-55 generating metadata which describes the multimedia data. Srivastava teaches in Fig. 
2 and col. 4 lines 8-13 translating metadata into a character-based text annotation. What 
Srivastava does not teach is inserting the annotation into the web page to form an enhanced web 
page suitable for processing by a character-based text processing system. Sidana does teach in 
Fig. 5 and col. 2 lines 13-30 inserting the annotation into the web page to form an enhanced web 
page suitable for processing by a character-based text processing system. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sidana into Srivastava to create the claimed invention. It would have been 
desirable to insert the annotation with the web page from which the multimedia file came to 
create an augmented, or enhanced, web page which could be more easily understood by the user. 
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This enhanced web page would decrease the amount of time required for the user to get the 
information from the web page which they need. 

Regarding dependent claim 11, which is dependent on claim 10, Srivastava and Sidana 
teach the limitations of claim 10 as explained above. Srivastava teaches identifying markup tags 
in a web page in Fig. 1 . The multimedia file obviously is contained with markup tags on the web 
page. Srivastava also teaches extracting the uniform resource locator of one of the external 
resources from at least one of the selected mark up tags in col. 3 lines 63-67. 

Regarding dependent claim 12, which is dependent on claim 10, Srivastava and Sidana 
teach the limitations of claim 10 as explained above. Srivastava teaches retrieving multimedia 
data from one or more external resources in Fig. 1 and analyzing the content of the multimedia 
data to extract metadata therefrom in col. 2 lines 46-55. 

Regarding dependent claim 13, which is dependent on claim 12, Srivastava and Sidana 
teach the limitations of claim 12 as explained above. Srivastava teaches identifying the data type 
of the multimedia data from each resource in col. 4 lines 32-35 and selecting a process routine 
for multimedia of the identified data type from each of said resources in col. 4 lines 39-48. 

Regarding dependent claim 14, which is dependent on claim 10, Srivastava and Sidana 
teach the limitations of claim 10 as explained above. Srivastava teaches indexing what may be 
an enhanced web page in light of claim 10 rejection to provide access to the web page in 
response to queries expressing one or more attributes expressed in the text annotation in col. 8 
lines 49-52. 

Regarding dependent claim 15, which is dependent on claim 10, Srivastava and Sidana 
teach the limitations of claim 10 as explained above. Srivastava teaches searching the content of 
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what may be an enhanced web page in light of claim 10 rejection in response to a search request 
to determine if attributes expressed in said search request are contained in the text annotation in 
col. 8 lines 49-52. 

Regarding dependent claim 16, which is dependent on claim 13, Srivastava and Sidana 
teach the limitations of claim 13 as explained above. Srivastava teaches indexing what may be 
an enhanced web page in light of claim 10 rejection to provide access to the web page in 
response to queries expressing one or more attributes expressed in the text annotation in col. 8 
lines 49-52. 

Regarding dependent claim 17, which is dependent on claim 13, Srivastava and Sidana 
teach the limitations of claim 13 as explained above. Srivastava teaches searching the content of 
what may be an enhanced web page in light of claim 10 rejection in response to a search request 
to determine if attributes expressed in said search request are contained in the text annotation in 
col. 8 lines 49-52. 



5. Applicant's arguments filed 09/23/2003 have been fully considered but they are not 
persuasive. 

6. Regarding Applicant's argument on page 2, paragraph 2, lines 1-4 that Srivistava et al. 
(hereafter referred to as Srivistava) does not suggest or describe means for analyzing a Web page 
to identify a markup tag which identifies an external media file which then described by 
metadata, the Examiner additionally cited fig. 1, not discussed by Applicant, as evidence for 
identification of a markup tag which identifies an external media file. Please note that Srivistava 



Response to Arguments 
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fig. 1 discloses a Media File, which could be an image, audio, or video file, which is derived 
from an internet resource. One skilled in the art at the time of the invention would have known 
that one of the most common ways to transmit image, audio, and video files through the internet 
is by use of HTML web pages viewable by internet web browsers. One of ordinary skill in the 
art at the time of the invention would also have known that HTML identifies image, audio, and 
video media files through the use of tags. Thus, the identification of markup tags which 
identifies and external media file is inherent to the Media File extraction from the internet 
disclosed in fig. 1 . 

7. Regarding Applicant's argument on page 2, paragraph 5 Sidana does teach combining a 
Web page and an annotation to form an enhanced Web page in fig. 5 and col. 2 lines 13-30. 
Metadata is a character-based text description and the Sidana annotations are character-based 
text descriptions as well, which are what is shown in fig. 7 item 712 and thus Sidana 
encompasses and can handle combining metadata annotations with a Web page. 

8. Regarding Applicant's argument on page 3, paragraphs 2-3, Srivistava is concerned with 
creating annotation describing the media files in col. 4 lines 8-13 and Sidana is concerned with 
taking annotations and combining them with Web pages so that the user may view an enhanced 
Web page. Srivistava fig. 1 is very similar to Applicant's fig. 1 in that the external media files 
are derived from an internet resource, and then the metadata is extracted from the media files. 
The next step is where they diverge with Srivistava placing the metadata annotations in a 
database and Applicant placing the annotations into an enhanced web page. To one of ordinary 
skill in the art at the time of the invention, Sidana is the teaching which takes the annotations of 
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Srivistava and incorporates them into an enhanced Web page as Applicant does, thus rendering 
Applicant's invention obvious. 

Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Peter J Smith whose telephone number is 703-305-5931. The 
examiner can normally be reached on Mondays-Fridays 7:00am-3 :30pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph H Feild can be reached on 703-305-9792. The fax phone number for the 
organization where this application or proceeding is assigned is (703) 872-9306. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-305-3900. 
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September 29, 2003 
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